Image sensor with interleaved hold for single-readout depth measurement

ABSTRACT

Time-of-flight (TOF) systems and techniques whereby a first exposure obtains pixel measurements for a first subset of pixels of a pixel array, using a first reference signal. For a second exposure, the first subset of the pixels, e.g., every second line of the pixel array, are set to a “hold” state, so that values obtained from the first measurement are maintained. A second exposure using a second reference signal is performed for a second subset of the pixels. The first and second reference signals may have different phase shifts relative to a signal modulating an optical signal being measured. The result is an array of pixels in which the first and second subsets hold results of the first and second exposures, respectively. These pixel values can then be read out all at once, with certain calculations being performed directly as pixel values are read from the pixel array.

BACKGROUND

In optical sensing applications, depth measurements, i.e., measurementsof the distance to various features of an object or objects in view ofan image sensor may be performed as so-called time-of-flight (ToF)measurements, which are distance measurements determined using the speedof light and image/pixel sensors. The distance to an object of interestis typically calculated per pixel and, once calculated, can be used fordepth detection, gesture identification, object detection, and the like.The distance per pixel is combined to create a depth map that provides athree-dimensional image.

Conventional approaches to TOF measurements require multiple sequentialexposures, also referred to as copies. Each exposure requires that lightgenerated from a light source be amplitude modulated at a respectivephase, the phase being different for different exposures. For example,one approach requires four separate exposures, such as 0°, 90°, 180° and270°. Measurement information from the four exposures is collected andcompared to determine a depth map. For high-precision measurements, withextended unambiguous ranges, even more exposures, e.g., as many as nineseparate raw measurements, may be performed.

With these approaches, a large amount of information needs to be storedand processed—for example, if nine separate exposures are performed,nine measurements for each pixel must be stored and processed, typicallyby an application processor (AP) separate from the integrated circuitcontaining the image sensor, with the stored multiple exposures thenbeing used to generate the depth map. While this approach allows veryprecise and detailed distance measurements, large amounts of storage,processing power, and power are needed.

In some applications, high-precision measurements are not needed, or arenot always needed. Accordingly, what is needed are techniques todetermine distances using the speed of light with reduced complexity andpower consumption.

SUMMARY

Time-of-flight (TOF) systems and techniques addressing these needs aredisclosed, whereby a first exposure obtains pixel measurements for atleast a first subset of the pixels of a pixel array, e.g., for aparticular phase shift of a reference signal relative to a signalmodulating an optical signal being measured. For a second exposure, thefirst subset of the pixels, e.g., every second line of the pixel array,are set to a “hold” state, so that the values obtained from the firstmeasurement are maintained. A second exposure, which may use a differentphase shift for the reference signal, relative to the modulated opticalsignal, is performed for a second subset of the pixels, e.g., all ofthose pixels not set to the hold state. The result is an array of pixelsin which the first and second subsets hold the results of the first andsecond exposures, respectively. These pixel values can then be read outall at once, with certain calculations being performed directly as pixelvalues are read from the pixel array.

Embodiments of the disclosed techniques include an example method forprocessing pixel signal values from an image sensor having a pluralityof pixels, where each pixel is configured to generate a respective pixelsignal value by demodulating received light using a reference signal.This example method includes controlling at least a first subset of theplurality of pixels to generate respective pixel signal values, based ona first reference signal applied to the pixels, for a first exposure.The first subset of the plurality of pixels to are then controlled tohold their respective pixel signal values. While the first subset of theplurality of pixels are holding their respective pixel signal values, asecond subset of the plurality of pixels are controlled to generaterespective pixel signal values, based on a second reference signalapplied to the pixels, for a second exposure. The first and secondreference signals may have different phases, with respect to a signalmodulating an optical signal being measured. This second subset of theplurality of pixels are then controlled to hold their respective pixelsignal values. The example method further comprises subsequently readingout the pixel signal values from (at least) the first and second subsetsof the plurality of pixels and converting the read pixel signal valuesto digital pixel values. In some embodiments, pixel signal values fromthe first subset of pixels may be combined with pixel signal values forcorresponding pixels in the second subset of pixels to obtain a distancemeasurement.

An example image processing system, according to some embodiments,includes a sensor that comprises a plurality of pixels configured togenerate a respective plurality of pixel signal values in response toreceived light, where each pixel is configured to obtain its respectivepixel signal value by demodulating received light using a referencesignal. This received light may be reflected from a target scene, forexample. The example image processing system further includes areference signal generator, which is configured to generate a referencesignal with a selectable phase, relative to the phase of a modulationsignal applied to light transmitted towards target scene, and to providethe reference signal to the plurality of pixels in sensor. The imageprocessing system still further includes an analog-to-digital converter(ADC) circuit, which may include one or several ADCs, operativelycoupled to the plurality of pixels in sensor.

The example image processing system further includes control circuitry,which may comprise, for example a processor, controller, or the like,and/or other digital logic. In several embodiments, the controlcircuitry is configured to cause the image processing system to carryout a method like the one summarized above. Thus, for example, thecontrol circuity may be configured to control the reference signalgenerator to generate a first reference signal, and to control at leasta first subset of the plurality of pixels in the sensor to generaterespective pixel signal values, based on the first reference signal, fora first exposure. The control circuitry may be further configured to,subsequent to the first exposure, control the first subset of theplurality of pixels to hold their respective pixel signal values, and tothen control the reference signal generator to generate a secondreference signal, while the first subset of the plurality of pixels areholding their respective pixel signal values. The first and secondreference signals may have different phases, with respect to a signalmodulating an optical signal being measured by the image sensor.

The control circuitry in these embodiments is still further configuredto, while the first subset of the plurality of pixels continue to holdtheir respective pixel signal values, control a second subset of theplurality of pixels to generate respective pixel signal values, based onthe second reference signal, for a second exposure, and then subsequentto the second exposure, control a second subset of the plurality ofpixels to hold their respective pixel signal values. Finally, thecontrol circuitry 61 is configured to read out the pixel signal values,for conversion to respective digital pixel values by the ADC circuit.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a system for time of flight measurementaccording to some of the embodiments described herein.

FIG. 2 illustrates an example photonic mixing device (PMD).

FIG. 3 is a diagram illustrating principles of phase measurementaccording to time-of-flight (TOF) techniques.

FIG. 4 is a schematic diagram illustrating an example arrangement ofpixels and read-out circuitry.

FIG. 5 is a flow diagram illustrating a method for processing pixelsignal values from an image sensor, according to some embodiments.

FIG. 6 is a block diagram illustrating components of an example imageprocessing system, according to some embodiments.

DETAILED DESCRIPTION

The present invention will now be described with reference to theattached drawing figures, wherein like reference numerals are used torefer to like elements throughout, and wherein the illustratedstructures and devices are not necessarily drawn to scale. In thisdisclosure, the terms “image” and “image sensor” are not limited toimages or sensors involving visible light but encompass the use ofvisible light and other electromagnetic radiation. Thus, the term“light” as used herein is meant broadly and refers to visible light aswell as infrared and ultraviolet radiation.

FIG. 1 illustrates the basic principles of continuous-wave (CW)time-of-flight (TOF) measurements, which are well known. A light source110, such as a light-emitting diode (LED) or vertical-cavitysurface-emitting laser (VCSEL), is modulated with an electrical signal(e.g., a radio-frequency sinusoid at, for example, 300 MHz), so that thelight source 110 emits an amplitude-modulated optical signal towards thetarget scene 120. Traveling at the speed of light c, the light signalreflects from an object or objects in the scene 120 and arrives back ata pixel array 135 in the TOF sensor 130, with the time of flight to thetarget scene 120 and back imposing a phase shift of ∅ on the opticalsignal as received at the pixel array 135, relative to the originallytransmitted optical signal.

The modulation signal 137 used to modulate the emitted light, or aphase-shifted version of it, is also supplied as a reference signal tothe pixels in pixel array 135, to be correlated with the modulationsignal superimposed on the reflected optical signal—in effect, thereflected optical signal is demodulated by each pixel in the pixel array135.

While the structure and design of light-sensing pixels may vary, each ofthe pixels in pixel array 135 may in some instances be a photon mixingdevice, or PMD. FIG. 2 illustrates the basic structure of an examplePMD, which includes readout diodes A and B and modulation gates A and B.A reference signal is applied differentially across the modulation gatesA and B, creating a gradient in electric potential across thep-substrate, while incoming light is received at a photo gate/diode. Adifferential sensor signal is generated across the readout diodes A andB. The sensor signal from a pixel may be integrated for a period of timeto determine phase measurement information.

The difference between voltages at the Read-A and Read-B nodes of thePMD corresponds to the correlation between the modulated optical signaldetected by the photosensitive diode structures in the illustrateddevice and the reference signal, which is applied between the Mod-A andMod-B nodes of the device. Thus, the PMD (and other light-sensitivepixel structures) demodulate the modulated optical signal reflected fromthe target scene 120, producing a pixel signal value (in this case thedifference between voltages at Read-A and Read-B) indicative of thedistance traveled by the reflected optical signal, as discussed infurther detail below.

While the modulation signal may take any of a variety of forms, theprinciple behind this correlation/demodulation is most easily seen witha sinusoidal signal as a modulation signal. If the modulation signalg(t) and the received signal s(t) with modulation amplitude ‘a’ andphase shift ‘∅’ are given as:m(t)=cos (ωt), ands(t)=1+(a×cos (ωt+∅)),then the correlation of the received signal with the reference signalgives:

${{r(\tau)} = {\left( \frac{a}{2} \right){\cos\left( {\varnothing + {w\;\tau}} \right)}}},$which is a function of the phase difference ∅ between the two signals.It will be appreciated that with a periodic modulation signal, thiscorrelation can be carried out for an extended period of time, e.g.,several cycles of the modulating signal, to improve the signal-to-noiseratio of the resulting measurement.

The phase difference between the emitted optical signal and the receivedreflection of that signal, which is proportional to the distancetraveled by the optical signal, can be extracted by an N-phase shiftingtechnique. This requires sampling the correlation function at Ndifferent points, e.g., by performing correlations using N differentphase shifts of the reference signal, with respect to the modulatingsignal g(t). At least two measurements are required to calculate thisphase shift, and hence to determine the distance traveled. This is oftendone using four different phase shifts, at 0, 90, 180, and 270 degrees,as this allows for a simple cancellation of systematic offsets in thecorrelation results. This is seen in FIG. 3, which shows how thecorrelations A0 and A1, at 0 and 90 degrees, respectively, correspond toa first phase vector having an “ideal” component corresponding to theactual difference traveled by the optical signal and a systematiccomponent reflecting systematic error in the measurements and readout.Likewise, the correlations A2 and A3, at 180 and 270 degrees,respectively, correspond to a second phase vector pointing in theopposite direction, with an exactly opposite “ideal” component and anidentical systematic component. In the figure, the ideal components arerepresented by the vectors extending from the origin to the circle,while the systematic error components are represented by the smallervectors. The actual phase ∅ can then be calculated as follows:

$Ø = {{\arctan\left( \frac{{A\; 1} - {A\; 3}}{{A\; 2} - {A\; 0}} \right)}.}$

From this phase, the distance, or “depth” to the target scene 120 can becalculated as follows:

${D = \frac{c \times \varnothing}{4\;\pi\; f_{mod}}},$where f_(mod) is the frequency of the modulating signal. It will beappreciated that because of “phase wrapping,” this distance calculationhas an ambiguous result, as it is not possible to tell from a singledistance calculation whether the distance traveled is less than a singlewavelength of the modulating waveform, or multiple wavelengths. Varioustechniques for resolving this ambiguity are well known, e.g., byincorporating amplitude information obtained from the reflected opticalsignal, and/or repeating the measurement with a different modulatingfrequency, but a detailed discussion of these techniques is unnecessaryto a full understanding of the presently disclosed techniques and isthus beyond the scope of the present disclosure.

In the calculation above, subtracting A3 from A1 and A0 from A2 cancelsthe systematic error. Thus, four or more phase measurements are commonlymade for each pixel of the pixel array 135, and combined, e.g., as shownin the phase calculation above, to result in a single distancemeasurement for each pixel. The resulting array of distance measurementsthus forms a “depth map” mapping the distances between the pixels andthe object or objects in the illuminated scene. However, the accuracyprovided by four or more phase measurements per depth value is notalways necessary. Further, the use of several measurements per depth mapis expensive, in terms of memory usage and processing resources, as thepixel signal value for each pixel in the pixel array 135 must be readfrom the array and stored several times, with each of these read-outscorresponding to a so-called exposure of the pixel array 135, beforebeing combined into depth map values.

Embodiments of the techniques and devices detailed below address thisproblem by providing a technique whereby a pixel array is subjected totwo (or more) exposures, corresponding to two (or more) phases of thereference signal, relative to the modulated optical signal emitted bythe TOF device, but where the pixel signal values are read-out only asingle time, reducing memory demands.

The approach is most easily explained with respect to a two-exposure(two-phase) implementation, although it will be appreciated that thetechnique can be extended to three, four, or more exposures. In anexample two-exposure implementation of the technique, the entire pixelarray is subjected to a first exposure to the reflected optical signal,with a first phase setting for the reference signal, relative to thesignal used to modulate the transmitted optical signal. After this firstexposure every second line of pixels is set into a “hold” state, suchthat it retains the pixel signal value, i.e., the demodulated outputrepresenting the correlation between the modulation on the receivedoptical signal and the reference signal. Pixels in a typical pixel arrayalready have this hold state, but, typically, the hold signal isglobally applied to all pixels, such that all pixels are in the holdstate, or not.

Unlike in conventional approaches to TOF measurements, after this firstexposure, no readout of the pixel array is immediately performed.Instead all the other pixels (which are not in hold state) are reset,and another exposure is done, with a second phase setting for thereference signal, relative to the modulation signal applied to thetransmitted optical signal. While this second exposure is performed, thefirst set of pixels, in every second line, are maintained in hold state,thus preserving the information from the first exposure, for thosepixels.

After the second exposure, the remaining pixels are set into hold state.The result is an array of pixels which alternately store, from one lineto the next, the result of the first or the second measurement. Thearray can then be read out a single time, effectively providing twophase measurements in one readout. In some embodiments, as discussedbelow, a distance calculation can be done on the sensor, by combiningtwo neighboring pixels, so as to produce a half-resolution depth mapfrom a single readout of the pixel array.

With this approach it is possible to do a dual-exposure, single-readoutmeasurement, with an on-chip distance calculation that does not requirethe involvement of an off-board application processor. In someimplementations, this may be a special mode of operation, with reducedaccuracy compared to a traditional nine-phase measurement, but withsignificantly reduced power consumption.

The two-exposure approach described above can be generalized to three ormore exposures. More generally, the presently described techniquesinclude controlling the pixels of an image sensor so that the pixelsignal values captured by a first subset of the pixels during the firstexposure are held, during one or more subsequent exposures that generatenew pixel signal values for the remaining subset of pixels.Subsequently, the pixel signal values for all of the pixels can be readin a single read-out of the pixel array, with the first subset yieldingpixel signal values corresponding to the first exposure, and hence thefirst phase of the reference signal, and the remaining pixels yieldingpixel signal values corresponding to the second (or subsequent)exposure, and hence the second (or other) phase of the reference signal.

It will be appreciated that in some embodiments of the techniquedescribed above, the second subset (and any additional subsets) ofpixels may generate pixel signal values during the first exposure, withthese pixel signal values effectively being discarded during the secondexposure. In other words, the second subset of pixels in theseembodiments operate normally during the first exposure, with theiroutput values being replaced by values from the second exposure beforebeing read. In other embodiments, however, the second subset of pixelsmay be deactivated or be otherwise configured to not generate pixelsignal values during the first exposure.

It will be further appreciated that the first and second (and anyadditional) subsets of pixels may comprise all of the pixels of an imagesensor or imaging device, in some embodiments, while comprising onlysome of the pixels of the sensor or imaging device, in others. In otherwords, in some devices, a first portion of a device's pixels may becontrollable according to the techniques described herein, while asecond portion of the device's pixels are configured for operationaccording to conventional techniques.

As discussed in further detail below, in some embodiments the firstsubset of pixels corresponds to every second row in an array of pixelshaving rows and columns. When the pixel signal values are read from thearray in these embodiments, each pixel in this first subset produces apixel signal value corresponding to the first exposure, while itsclosest neighboring pixel in an adjacent row produces a pixel signalvalue corresponding to the second exposure. If the first and secondexposures correspond to a 90-degree difference in the respectivereference signal phases, the pixel signal value for each pixel in thefirst subset can be immediately combined with the pixel signal value forits neighboring pixel in the adjacent row, e.g., according to

${\varnothing = {\arctan\left( \frac{A\; 1}{A\; 0} \right)}},$to obtain a phase value, for direct application to a depth map. As alsodiscussed in further detail below, in some embodiments of the devicesdisclosed herein, this calculation is performed by logic implementeddirectly on the same chip incorporating the pixel array, and can beimplemented so as to further reduce storage/memory requirements.

FIG. 4 is a schematic diagram illustrating elements of an example imageprocessing system incorporating aspects of the techniques discussedabove. It should be understood that FIG. 4 includes a simplified drawingof a pixel array, only illustrating four lines of pixels of what may bea much larger array of pixels, together with a subset of thecorresponding readout circuitry. Furthermore, it should be understoodthat the illustrated pixels are not PMD pixels, but standard pixels, tosimplify the drawing. However, the concept can be directly applied toPMD pixels or to other pixel structures that demodulate a modulatedlight signal so as to produce a correlation between the modulation onthe light signal and a reference signal provided to the pixels.

The lines extending into the pixel array from the left side are the holdlines. In a conventional implementation of a pixel array, the hold linesfor all rows are connected to a single global hold signal, whichcontrols the hold switches of all the pixels synchronously. In theillustrated solution, however, the global hold signal is split in two,such that neighboring rows of pixels can be controlled independently.This has no effect on the pixel array itself—only the distribution ofthe control signals outside the array needs to be modified, compared toconventional pixel array structures.

In the schematic of FIG. 4, there are also so-called row-select signals,labeled row_0, row_1, row_2 and row_3 in the figure. During a readoutout of the pixel array, one of the row_select signals at a time isactive, which means that there is always one line of pixels connected tothe analog-to-digital converters (ADCs), shown on the right of FIG. 4.When analog to digital conversion of the selected row is finished, thenext row is selected, and so on. While it is possible to use only asingle ADC, that is sequentially connected to the pixel in each columnof the selected row, the illustrated implementation uses 16 ADCs, withthe 224 columns in this example circuit being divided into groups of 14.With this approach, 16 columns at a time are read, and only 14sequential read operations.

In some embodiments of a pixel array like that shown in FIG. 4, twoblocks of pixel memory are used in ping-pong operation. While the ADC isconverting one line of pixels and storing the data in one of the memoryblocks, a high-speed data interface (not shown) transfers data from theother memory block to the application processor. When both of theseprocesses are finished (conversion of one line and transmission ofanother line), the operation switches, meaning that the data which havejust been generated in the one memory block is now transferred via theinterface and the other memory block is used for conversion.

In some embodiments, logic for calculating an arctangent (ATAN) from twopixel signal values before transmission is added. Advantageously, thismay be implemented on the same integrated circuit device that carriesthe array of pixels, i.e., as a function that is “on-board” the imagesensor device rather than in a separate processor. Implementation of anATAN function can be done using very efficiently, using algorithms likethe so-called “CORDIC” algorithm. It will be appreciated that thisfurther reduces the memory requirements—with this approach, a first rowof pixel signal values is read from the pixel signal array, convertedinto digital form, and stored in pixel memory. Then, the adjacent row ofpixel signal values is read and digitized—as each of these values isread, it can be immediately combined with the stored value for theneighboring value, and transferred off chip. Thus, half-resolution depthmap values are directly delivered by the pixel array circuit to anoff-board memory, in some embodiments.

With the above explanations in mind, it will be appreciated that FIG. 5illustrates an example method for processing pixel signal values from animage sensor having a plurality of pixels, where each pixel isconfigured to generate a respective pixel signal value by demodulatingreceived light using a reference signal. As shown in block 510, themethod includes controlling at least a first subset of the plurality ofpixels to generate respective pixel signal values, based on a firstreference signal applied to (at least) the first subset of the pixels,for a first exposure. As shown in block 520, the first subset of theplurality of pixels to are then controlled to hold their respectivepixel signal values. In some embodiments, pixel signal values aregenerated for all of the pixels of the pixel array during this firstexposure, but the pixel measurements for pixels other than the firstsubset are not maintained—i.e., these pixel signal values are not “held”after the first measurement.

While the first subset of the plurality of pixels are holding theirrespective pixel signal values, a second subset of the plurality ofpixels are controlled to generate respective pixel signal values, basedon a second reference signal applied to (at least) the second subset ofthe pixels. This is illustrated in block 530. As shown at block 540,this second subset of the plurality of pixels are then controlled tohold their respective pixel signal values.

As discussed with respect to the detailed examples provided above, thefirst reference signal may have a first phase, relative to a modulatingsignal applied, during the first exposure, to an optical signal beingmeasured by the image sensor, while the second reference signal, whichmay otherwise be identical to the first reference signal, has a secondphase, relative to the modulating signal applied, during the secondexposure, to the optical signal being measured by the image sensor. Itwill be appreciated that this difference in phase between the first andsecond reference signals may be generated by introducing a phase shiftto an input reference signal, between the first and second exposures, orby introducing a phase shift to the modulating signal applied to theoptical signal under measurement, between the first and secondexposures, or by a combination of both.

While not shown in FIG. 5, this technique can be extended to a third,fourth, or more subsets of pixels, corresponding to respective third,fourth, and subsequent phases. In any case, the illustrated methodfurther comprises subsequently reading out the pixel signal values from(at least) the first and second subsets of the plurality of pixels andconverting the read pixel signal values to digital pixel values, asshown at block 550. It should be further understood that the imagesensor may, in some embodiments, have further pixels, in addition tothose that are controlled according to the techniques described herein.

In some embodiments, the plurality of pixels are arranged in an arrayhaving lines and columns, e.g., where the first subset of pixelscomprises pixels in every odd or even line of the array and the secondsubset of pixels comprises the remaining pixels. As discussed above,this makes it especially convenient to perform on-board phasecalculations, using neighboring pixels that correspond to differentexposures with different phases for the reference signals applied duringthe exposures. Thus, in some embodiments of the illustrated method, themethod may further include generating depth values for a depth map bycombining, for each depth value, a digital pixel value for a pixel ofthe first subset with a digital pixel value for a corresponding pixel ofthe second subset. This is shown at block 560 of FIG. 5, which isillustrated with a dashed outline to indicate that it need not bepresent in every embodiment or instance of the illustrated method. Insome of these embodiments, the plurality of pixels are arranged in anarray having lines and columns, the first subset of pixels comprisingpixels in every odd or even line of the array and the second subset ofpixels comprising the remaining pixels, and wherein said combiningcomprises, for each depth value, combining a digital pixel value for apixel in any given line with the digital pixel value for the neighboringpixel in an immediately adjacent line. In some embodiments, thedifference between the phase of the first reference signal and the phaseof the second reference signal, relative to the signal modulating theoptical signal under measurement, is 90 degrees, in which case saidcombining may comprise computing an arctangent of a ratio of the digitalpixel values for the pixels.

FIG. 6, correspondingly, illustrates an example image processing system600, according to several embodiments of the presently disclosed devicesand systems. The system 600 can be utilized to detect objects, e.g., asshown in target scene 602, as well as to determine distances to thedetected objects. System 600 may be a continuous-wave TOF system, suchas a photon modulation device (PMD)-based TOF system.

The illustrated system 600 includes a light source 624, which isconfigured to amplitude modulate a beam of light and emit theamplitude-modulated light towards the scene 602. The amplitudemodulation may be based on a reference signal generated by referencesignal generator 608. The reference signal may be a radio-frequency (RF)signal, e.g., in the MHz range, although other modulation frequenciescan be used. The emitted light can include light having varied ranges ofwavelength, such as sunlight and infra-red. The emitted light reflectsfrom one or more objects in the scene and returns to the sensor 604.

Image process system 600 may be configured to carry out a techniquewhereby a first exposure obtains pixel measurements for at least a firstsubset of the pixels of a pixel array at a single interval of time, fora particular phase shift of the reference signal, relative to a signalmodulating the optical signal under measurement. For a second exposure,each of the first subset of the pixels is set to a “hold” state, so thatthe values obtained from the first measurement are maintained. Thisfirst subset may be, for example, every second line of the pixel array.In some embodiments, pixel measurements are obtained for all of thepixels of the pixel array during the first measurement/exposure, but thepixel measurements for pixels other than the first subset are notmaintained—i.e., these pixels are not “held” after the firstmeasurement. A second exposure, using a different phase (e.g., with a90-degree shift, relative to the first exposure) is performed for asecond subset of the pixels, e.g., all of those pixels that were not setto the hold state after the first exposure.

The result of this process is an array of pixels in which the firstsubset of pixels holds the results of the first exposure, while thesecond subset of pixels holds the results of the second exposure. Thesepixel values can be read out all at once, after the second exposure,with certain calculations being performed directly, as the pixel valuesare read from the pixel array. This approach can dramatically reduce theneed for data storage, as separate memory for storing pixel valuesbetween exposures is not needed. Further, this approach can simplify theprocessing requirements for generating a depth map, as an on-chipdistance calculation can be performed between neighboring pixels as thepixel values are read from the array. Thus, memory, complexity, andpower consumption are reduced.

Referring back to FIG. 6, the illustrated image processing system 600includes a sensor 604, which comprises a plurality of pixels configuredto generate a respective plurality of pixel signal values in response toreceived light 614, where each pixel is configured to obtain itsrespective pixel signal value by demodulating received light using areference signal 622. As seen in FIG. 6, received light 602 may bereflected from a target scene 602. As discussed above, while severalsuitable pixel configurations are possible, one suitable pixel design isthe PMD described above.

The numbers of pixels, rows, and columns can vary, from one embodimentto another, and are selected based on factors including desiredresolution, intensity, and the like. In one example, these sensorcharacteristics are selected based on the objects to be detected and theexpected distances to the objects. Thus, for example, the pixelresolution of the pixels in sensor 604 may vary, from one embodiment toanother. Small objects require a higher resolution for detection. Forexample, finger detection requires a resolution of <5 mm per pixel at adistance or range of about 0.5 meters. Medium sized objects, such ashand detection, require a resolution of <20 mm per pixel at a range ofabout 1.5 meters. Larger sized objects, such as a human body, require aresolution of <60 mm per pixel at about 2.5 meters. It is appreciatedthat the above examples are provided for illustrative purposes only andthat variations can occur including other objects, resolutions anddistances for detection. Some examples of suitable resolutions includeVGA—640×400 pixels, CIF—352×288 pixels, QQ-VGA—160×120 pixels, and thelike.

Image processing system 600 further includes a reference signalgenerator 608, which may be configured, in some embodiments, to generatereference signal 622 with a selectable phase, relative to the phase of amodulation signal applied to light transmitted towards target scene 602,and to provide the reference signal 622 to the plurality of pixels insensor 604. Image processing system 600 still further includes ananalog-to-digital converter (ADC) circuit 606, which may include one orseveral ADCs, operatively coupled to the plurality of pixels in sensor604.

The illustrated image processing system 600 further includes controlcircuitry 612, which may comprise, for example a processor, controller,or the like, and/or other digital logic. In several embodiments, thecontrol circuitry 612 is configured to cause the image processing system600 to carry out a method like those described above, in connection withFIG. 5. Thus, for example, control circuity 612 may be configured tocontrol the reference signal generator 608 to generate a first referencesignal, and to control at least a first subset of the plurality ofpixels in sensor 604 to generate respective pixel signal values, basedon the first reference signal, for a first exposure. Control circuitry612 is further configured to, subsequent to the first exposure, controlthe first subset of the plurality of pixels to hold their respectivepixel signal values, and to then control the reference signal generator608 to generate a second reference signal, while the first subset of theplurality of pixels are holding their respective pixel signal values.

The control circuitry 612, in some embodiments, is configured togenerate the first reference signal so that it has a first phase,relative to a modulating signal applied, during the first exposure, toan optical signal being measured by the image sensor, and to generatethe second reference signal so that it has a second phase, relative tothe modulating signal applied, during the second exposure, to theoptical signal being measured by the image sensor. This may be done, forexample, by introducing a phase shift to an input reference signal,between the first and second exposures, at a point before the referencesignal is applied to the pixels.

Control circuitry 612 in these embodiments is still further configuredto, while the first subset of the plurality of pixels continue to holdtheir respective pixel signal values, control a second subset of theplurality of pixels to generate respective pixel signal values, based onthe second reference signal, for a second exposure, and then subsequentto the second exposure, control the second subset of the plurality ofpixels to hold their respective pixel signal values. Finally, controlcircuitry 612 is configured to read out the pixel signal values, asshown at signal 618 in FIG. 6, for conversion to respective digitalpixel values by the ADC circuit 606.

In some embodiments of the image processing system 600 shown in FIG. 6,the plurality of pixels in sensor 604 may be arranged in an array havinglines and columns, where the first subset of pixels comprises pixels inevery odd or even line of the array and the second subset of pixelscomprises the remaining pixels. In some embodiments, the imageprocessing system 600 may further include calculating logic, representedin FIG. 6 by depth map generator 610, configured to generate depthvalues for a depth map by combining, for each depth value, a digitalpixel value for a pixel of the first subset with a digital pixel valuefor a corresponding pixel of the second subset. In some of theseembodiments, where the plurality of pixels are arranged in an arrayhaving lines and columns, the first subset of pixels comprising pixelsin every odd or even line of the array and the second subset of pixelscomprising the remaining pixels, the calculating logic may be configuredto, for each depth value, combine a digital pixel value for a pixel inany given line with the digital pixel value for the neighboring pixel inan immediately adjacent line. In these embodiments, the differencebetween the phase of the first reference signal and the phase of thesecond reference signal, relative to a signal modulating the opticalsignal under measurement, may be 90 degrees, for example, where thecalculating logic is configured to combine digital pixel values bycomputing an arctangent of a ratio of the digital pixel values for thepixels. this may be done, for example, using an implementation of thewell-known CORDIC algorithm, or using a look-up table, for example. Insome embodiments, image processing system 600 comprises a memoryoperatively coupled to the ADC circuit 606, e.g., as part of depth mapgenerator 610, and configured to store digital pixel values for one lineof the array at a time, where the calculating logic is configured tocompute each arctangent using a digital pixel value stored in the memoryand a corresponding digital pixel value for the neighboring pixel in theimmediately adjacent line.

In some embodiments, system 600 may be used for low power applicationsand modes, to mitigate power consumption. However, it is appreciatedthat the system 600 can be used for other applications and modes. It isalso appreciated that variations in the components of the system 100 arecontemplated, including additional components and/or omission of showncomponents.

In one variation, the system 600 uses the combined pixels, as describedabove to generate a depth map in the low power mode. However, in ahigh-resolution mode, the same system 600 may use multiple exposures toobtain a higher resolution depth map. In some embodiments, the system600 can selectively switch between the mode using combined pixels andthe mode using multiple exposures to obtain a higher resolution. Themode using combined pixels may be a low power mode with a lowerresolution when compared to the multiple exposure mode or approach.Furthermore, in the combined pixel mode, the system 600 may be capableto roughly calculate 3D data directly on the sensor chip without theneed for data processing external to the sensor chip, since as discussedabove, a rough distance calculation can be performed using only a simplearctan calculation. Such a calculation may be implemented fully inhardware directly on the sensor chip using gates and/or other logicavailable in hardware. Thus, the combined pixel mode may allowdetermining 3D data fully on the sensor chip without external calculatordevices.

The system 600 in some embodiments can switch from the lower power modedescribed herein to a higher resolution mode based on a number offactors or conditions. The low-power mode is also referred to as alow-resolution mode. For example, a touchless gesture control system canuse the lower power mode initially and switch to the higher resolutionmode once activity is detected to enable more precise measurements.Another example is face detection, where the lower power mode is usedfor shape detection. Once a shape is detected, the system switches tohigher resolution mode to measure facial features. Other variations anduses are also contemplated for the system 600.

In view of the detailed discussion above, it will be appreciated thatthe claimed subject matter may be implemented as a method, apparatus, orarticle of manufacture using standard programming and/or engineeringtechniques to produce software, firmware, hardware, or any combinationthereof to control a computer to implement the disclosed subject matter.The term “article of manufacture” as used herein is intended toencompass a computer program accessible from any computer-readabledevice, carrier, or media. Of course, those skilled in the art willrecognize many modifications may be made to this configuration withoutdeparting from the scope or spirit of the claimed subject matter.

In particular regard to the various functions performed by the abovedescribed components or structures (assemblies, devices, circuits,systems, etc.), the terms (including a reference to a “means”) used todescribe such components are intended to correspond, unless otherwiseindicated, to any component or structure which performs the specifiedfunction of the described component (e.g., that is functionallyequivalent), even though not structurally equivalent to the disclosedstructure which performs the function in the herein illustratedexemplary implementations of the invention. In addition, while aparticular feature of the invention may have been disclosed with respectto only one of several implementations, such feature may be combinedwith one or more other features of the other implementations as may bedesired and advantageous for any given or particular application.Furthermore, to the extent that the terms “including”, “includes”,“having”, “has”, “with”, or variants thereof are used in either thedetailed description and the claims, such terms are intended to beinclusive in a manner similar to the term “comprising”.

What is claimed is:
 1. A method for processing pixel signal values from an image sensor having a plurality of pixels, each pixel being configured to generate a respective pixel signal value by demodulating received light using a reference signal, the method comprising: controlling at least a first subset of the plurality of pixels to generate respective pixel signal values, based on a first reference signal applied to the first subset of pixels, for a first exposure; subsequent to the first exposure, controlling the first subset of the plurality of pixels to hold their respective pixel signal values; while the first subset of the plurality of pixels are holding their respective pixel signal values, controlling a second subset of the plurality of pixels to generate respective pixel signal values, based on a second reference signal applied to the second subset of pixels, for a second exposure, wherein the first and second subsets are mutually exclusive; subsequent to the second exposure, controlling the second subset of the plurality of pixels to hold their respective pixel signal values; and reading out a pixel signal value from each pixel of the first and second subsets of the plurality of pixels and converting each read pixel signal value to a respective digital pixel value; wherein the first reference signal has a first phase, relative to a modulating signal applied during the first exposure to an optical signal being measured by the image sensor, and the second reference signal has a second phase, relative to the modulating signal applied during the second exposure to the optical signal being measured by the image sensor.
 2. The method of claim 1, wherein the plurality of pixels are arranged in an array having lines and columns, and wherein the first subset of pixels comprises pixels in every odd or even line of the array and the second subset of pixels comprises the remaining pixels.
 3. The method of claim 1, wherein the method further comprises generating depth values for a depth map by combining, for each depth value, a digital pixel value for a pixel of the first subset with a digital pixel value for a corresponding pixel of the second subset.
 4. The method of claim 3, wherein the plurality of pixels are arranged in an array having lines and columns, the first subset of pixels comprising pixels in every odd or even line of the array and the second subset of pixels comprising the remaining pixels, and wherein said combining comprises, for each depth value, combining a digital pixel value for a pixel in any given line with the digital pixel value for the neighboring pixel in an immediately adjacent line.
 5. The method of claim 1, wherein the method further comprises generating depth values for a depth map by combining, for each depth value, a digital pixel value for a pixel of the first subset with a digital pixel value for a corresponding pixel of the second subset, wherein the difference between the first phase and the second phase is 90 degrees, and wherein said combining comprises computing an arctangent of a ratio of the digital pixel values for the pixels.
 6. An image processing system comprising: a sensor comprising a plurality of pixels configured to generate a respective plurality of pixel signal values in response to received light, wherein each pixel is configured to obtain its respective pixel signal value by demodulating received light using a reference signal; a reference signal generator configured to generate the reference signal and to provide the reference signal to the plurality of pixels, and an analog-to-digital converter (ADC) circuit operatively coupled to the plurality of pixels, and control circuitry configured to: control the reference signal generator to generate a first reference signal; control at least a first subset of the plurality of pixels to generate respective pixel signal values, based on the first reference signal, for a first exposure; subsequent to the first exposure, control the first subset of the plurality of pixels to hold their respective pixel signal values; control the reference signal generator to generate a second reference signal, while the first subset of the plurality of pixels are holding their respective pixel signal values; while the first subset of the plurality of pixels are holding their respective pixel signal values, control a second subset of the plurality of pixels to generate respective pixel signal values, based on the second reference signal, for a second exposure, wherein the first and second subsets are mutually exclusive; subsequent to the second exposure, control the second subset of the plurality of pixels to hold their respective pixel signal values; and, read out a pixel signal value from each pixel of the first and second subsets of pixels, for conversion to respective digital pixel values by the ADC circuit; wherein the control circuitry is configured to generate the first reference signal so that it has a first phase, relative to a modulating signal applied during the first exposure to an optical signal being measured by the image sensor, and to generate the second reference signal so that it has a second phase, relative to the modulating signal applied during the second exposure to the optical signal being measured by the image sensor.
 7. The image processing system of claim 6, wherein the plurality of pixels are arranged in an array having lines and columns, and wherein the first subset of pixels comprises pixels in every odd or even line of the array and the second subset of pixels comprises the remaining pixels.
 8. The image processing system of claim 6, wherein the image processing system further comprises calculating logic configured to generate depth values for a depth map by combining, for each depth value, a digital pixel value for a pixel of the first subset with a digital pixel value for a corresponding pixel of the second subset.
 9. The image processing system of claim 8, wherein the plurality of pixels are arranged in an array having lines and columns, the first subset of pixels comprising pixels in every odd or even line of the array and the second subset of pixels comprising the remaining pixels, and wherein the calculating logic is configured to, for each depth value, combine a digital pixel value for a pixel in any given line with the digital pixel value for the neighboring pixel in an immediately adjacent line.
 10. The image processing system of claim 6, wherein the control circuitry is configured to generate the second reference signal to have the second phase, during the second exposure, by introducing a phase shift to an input reference signal, between the first and second exposures.
 11. The image processing system of claim 6, wherein the image processing system further comprises calculating logic configured to generate depth values for a depth map by combining, for each depth value, a digital pixel value for a pixel of the first subset with a digital pixel value for a corresponding pixel of the second subset, wherein the difference between the first phase and the second phase is 90 degrees and wherein the calculating logic is configured to combine digital pixel values by computing an arctangent of a ratio of the digital pixel values for the pixels.
 12. The image processing system of claim 11, further comprising a memory operatively coupled to the ADC circuit and configured to store digital pixel values for one line of the array at a time, wherein the calculating logic is configured to compute each arctangent using a digital pixel value stored in the memory and a corresponding digital pixel value for the neighboring pixel in the immediately adjacent line.
 13. The method of claim 1, further comprising, for each pixel in the first subset of the plurality of pixels, combining the digital pixel value for the pixel in the first subset with a digital pixel value for a corresponding pixel of the second subset.
 14. The image processing system of claim 6, further comprising calculating logic configured to, for each pixel in the first subset of the plurality of pixels, combine the digital pixel value for the pixel in the first subset with a digital pixel value for a corresponding pixel of the second subset. 